Independent Factor Analysis 1 Statistical Modeling and Blind Source Separation

We introduce the independent factor analysis (IFA) method for recovering independent hidden sources from their observed mixtures. IFA generalizes and uniies ordinary factor analysis (FA), principal component analysis (PCA), and independent component analysis (ICA), and can handle not only square noiseless mixing, but also the general case where the number of mixtures diiers from the number of sources and the data are noisy. IFA is a two-step procedure. In the rst step, the source densities, mixing matrix and noise covariance are estimated from the observed data by maximum likelihood. For this purpose we present an expectation-maximization (EM) algorithm, which performs unsupervised learning of an associated probabilistic model of the mixing situation. Each source in our model is described by a mixture of Gaussians, thus all the probabilistic calculations can be performed analytically. In the second step, the sources are reconstructed from the observed data by an optimal non-linear estimator. A variational approximation of this algorithm is derived for cases with a large number of sources, where the exact algorithm becomes intractable. Our IFA algorithm reduces to the one for ordinary FA when the sources become Gaussian, and to an EM algorithm for PCA in the zero-noise limit. We derive an additional EM algorithm speciically for noiseless IFA. This algorithm is shown to be superior to ICA since it can learn arbitrary source densities from the data. Beyond blind separation, IFA can be used for modeling multi-dimensional data by a highly constrained mixture of Gaussians, and as a tool for non-linear signal encoding. In the blind source separation (BSS) problem one is presented with multi-variable data measured by L 0 sensors. It is known that these data arise from L source signals that are mixed together by some linear transformation corrupted by noise. It is further known that the sources are mutually statistically independent. The task is to obtain those source signals. However, the sources are not observable and nothing is known about their properties beyond their mutual statistical independence, nor about the properties of the mixing process and the noise. In the absence of this information, one has to proceed`blindly' to recover the source signals from their observed noisy mixtures. Despite its signal-processing appearance, BSS is a problem in statistical modeling of data. In this context, one wishes to describe the L 0 observed variables y i , which are generally correlated, in terms of a smaller set of L …